{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About this notebook\n",
    "\n",
    "In this notebook, I quickly explore the `biorxiv` subset of the papers. Since it is stored in JSON format, the structure is likely too complex to directly perform analysis. Thus, I not only explore the structure of those files, but I also provide the following helper functions for you to easily format inner dictionaries from each file:\n",
    "* `format_name(author)`\n",
    "* `format_affiliation(affiliation)`\n",
    "* `format_authors(authors, with_affiliation=False)`\n",
    "* `format_body(body_text)`\n",
    "* `format_bib(bibs)`\n",
    "\n",
    "Feel free to reuse those functions for your own purpose! If you do, please leave a link to this notebook.\n",
    "\n",
    "Throughout the EDA, I show you how to use each of those files. At the end, I show you how to generate a clean version of the `biorxiv` as well as all the other datasets, which you can directly use by choosing this notebook as a data source (\"File\" -> \"Add or upload data\" -> \"Kernel Output File\" tab -> search the name of this notebook).\n",
    "\n",
    "### Update Log\n",
    "\n",
    "* V9: First release.\n",
    "* V10: Updated paths to include the [14k new papers](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/discussion/137474)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "from pprint import pprint\n",
    "from copy import deepcopy\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unhide the cell below to find the definition of the following functions:\n",
    "* `format_name(author)`\n",
    "* `format_affiliation(affiliation)`\n",
    "* `format_authors(authors, with_affiliation=False)`\n",
    "* `format_body(body_text)`\n",
    "* `format_bib(bibs)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_kg_hide-input": true
   },
   "outputs": [],
   "source": [
    "def format_name(author):\n",
    "    middle_name = \" \".join(author['middle'])\n",
    "    \n",
    "    if author['middle']:\n",
    "        return \" \".join([author['first'], middle_name, author['last']])\n",
    "    else:\n",
    "        return \" \".join([author['first'], author['last']])\n",
    "\n",
    "\n",
    "def format_affiliation(affiliation):\n",
    "    text = []\n",
    "    location = affiliation.get('location')\n",
    "    if location:\n",
    "        text.extend(list(affiliation['location'].values()))\n",
    "    \n",
    "    institution = affiliation.get('institution')\n",
    "    if institution:\n",
    "        text = [institution] + text\n",
    "    return \", \".join(text)\n",
    "\n",
    "def format_authors(authors, with_affiliation=False):\n",
    "    name_ls = []\n",
    "    \n",
    "    for author in authors:\n",
    "        name = format_name(author)\n",
    "        if with_affiliation:\n",
    "            affiliation = format_affiliation(author['affiliation'])\n",
    "            if affiliation:\n",
    "                name_ls.append(f\"{name} ({affiliation})\")\n",
    "            else:\n",
    "                name_ls.append(name)\n",
    "        else:\n",
    "            name_ls.append(name)\n",
    "    \n",
    "    return \", \".join(name_ls)\n",
    "\n",
    "def format_body(body_text):\n",
    "    texts = [(di['section'], di['text']) for di in body_text]\n",
    "    texts_di = {di['section']: \"\" for di in body_text}\n",
    "    \n",
    "    for section, text in texts:\n",
    "        texts_di[section] += text\n",
    "\n",
    "    body = \"\"\n",
    "\n",
    "    for section, text in texts_di.items():\n",
    "        body += section\n",
    "        body += \"\\n\\n\"\n",
    "        body += text\n",
    "        body += \"\\n\\n\"\n",
    "    \n",
    "    return body\n",
    "\n",
    "def format_bib(bibs):\n",
    "    if type(bibs) == dict:\n",
    "        bibs = list(bibs.values())\n",
    "    bibs = deepcopy(bibs)\n",
    "    formatted = []\n",
    "    \n",
    "    for bib in bibs:\n",
    "        bib['authors'] = format_authors(\n",
    "            bib['authors'], \n",
    "            with_affiliation=False\n",
    "        )\n",
    "        formatted_ls = [str(bib[k]) for k in ['title', 'authors', 'venue', 'year']]\n",
    "        formatted.append(\", \".join(formatted_ls))\n",
    "\n",
    "    return \"; \".join(formatted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unhide the cell below to find the definition of the following functions:\n",
    "* `load_files(dirname)`\n",
    "* `generate_clean_df(all_files)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_kg_hide-input": true
   },
   "outputs": [],
   "source": [
    "def load_files(dirname):\n",
    "    filenames = os.listdir(dirname)\n",
    "    raw_files = []\n",
    "\n",
    "    for filename in tqdm(filenames):\n",
    "        filename = dirname + filename\n",
    "        file = json.load(open(filename, 'rb'))\n",
    "        raw_files.append(file)\n",
    "    \n",
    "    return raw_files\n",
    "\n",
    "def generate_clean_df(all_files):\n",
    "    cleaned_files = []\n",
    "    \n",
    "    for file in tqdm(all_files):\n",
    "        features = [\n",
    "            file['paper_id'],\n",
    "            file['metadata']['title'],\n",
    "            format_authors(file['metadata']['authors']),\n",
    "            format_authors(file['metadata']['authors'], \n",
    "                           with_affiliation=True),\n",
    "            format_body(file['abstract']),\n",
    "            format_body(file['body_text']),\n",
    "            format_bib(file['bib_entries']),\n",
    "            file['metadata']['authors'],\n",
    "            file['bib_entries']\n",
    "        ]\n",
    "\n",
    "        cleaned_files.append(features)\n",
    "\n",
    "    col_names = ['paper_id', 'title', 'authors',\n",
    "                 'affiliations', 'abstract', 'text', \n",
    "                 'bibliography','raw_authors','raw_bibliography']\n",
    "\n",
    "    clean_df = pd.DataFrame(cleaned_files, columns=col_names)\n",
    "    clean_df.head()\n",
    "    \n",
    "    return clean_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Biorxiv: Exploration\n",
    "\n",
    "Let's first take a quick glance at the `biorxiv` subset of the data. We will also use this opportunity to load all of the json files into a list of **nested** dictionaries (each `dict` is an article)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of articles retrieved from biorxiv: 1625\n"
     ]
    }
   ],
   "source": [
    "biorxiv_dir = '/kaggle/input/CORD-19-research-challenge/biorxiv_medrxiv/biorxiv_medrxiv/pdf_json/'\n",
    "filenames = os.listdir(biorxiv_dir)\n",
    "print(\"Number of articles retrieved from biorxiv:\", len(filenames))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_files = []\n",
    "\n",
    "for filename in filenames:\n",
    "    filename = biorxiv_dir + filename\n",
    "    file = json.load(open(filename, 'rb'))\n",
    "    all_files.append(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary keys: dict_keys(['paper_id', 'metadata', 'abstract', 'body_text', 'bib_entries', 'ref_entries', 'back_matter'])\n"
     ]
    }
   ],
   "source": [
    "file = all_files[0]\n",
    "print(\"Dictionary keys:\", file.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Biorxiv: Abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The abstract dictionary is fairly simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'cite_spans': [],\n",
      "  'ref_spans': [],\n",
      "  'section': 'Abstract',\n",
      "  'text': 'We model the extent to which age targeted quarantine can be used to '\n",
      "          'reduce ICU admissions caused by novel coronavirus COVID-19. Using '\n",
      "          'demographic data from New Zealand, we demonstrate that lowering the '\n",
      "          'age threshold for quarantine to 50 years of age reduces ICU '\n",
      "          'admissions drastically, and show that for sufficiently strict '\n",
      "          'isolation protocols, isolating one third of the countries '\n",
      "          'population for a total of 6 months is sufficient to avoid '\n",
      "          'overwhelming ICU capacity throughout the entire course of the '\n",
      "          'epidemic. Similar results are expected to hold for other countries, '\n",
      "          'though some minor adaption will be required based on local age '\n",
      "          'demographics and hospital facilities.'},\n",
      " {'cite_spans': [],\n",
      "  'ref_spans': [],\n",
      "  'section': 'Abstract',\n",
      "  'text': '. CC-BY 4.0 International license It is made available under a '\n",
      "          'author/funder, who has granted medRxiv a license to display the '\n",
      "          'preprint in perpetuity.'},\n",
      " {'cite_spans': [],\n",
      "  'ref_spans': [],\n",
      "  'section': 'Abstract',\n",
      "  'text': 'is the (which was not peer-reviewed) The copyright holder for this '\n",
      "          'preprint .'}]\n"
     ]
    }
   ],
   "source": [
    "pprint(file['abstract'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Biorxiv: body text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first probe what the `body_text` dictionary looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "body_text type: <class 'list'>\n",
      "body_text length: 51\n",
      "body_text keys: dict_keys(['text', 'cite_spans', 'ref_spans', 'section'])\n"
     ]
    }
   ],
   "source": [
    "print(\"body_text type:\", type(file['body_text']))\n",
    "print(\"body_text length:\", len(file['body_text']))\n",
    "print(\"body_text keys:\", file['body_text'][0].keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take a look at the first part of the `body_text` content. As you will notice, the body text is separated into a list of small subsections, each containing a `section` and a `text` key. Since multiple subsection can have the same section, we need to first group each subsection before concatenating everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "body_text content:\n",
      "[{'cite_spans': [{...}, {...}],\n",
      "  'ref_spans': [],\n",
      "  'section': 'Introduction',\n",
      "  'text': 'COVID-19, initially observed/detected in Hubei province of China '\n",
      "          'during December 2019, has since spread to all but a handful '\n",
      "          'countries, causing (as of the time of writing) an estimated 855,000 '\n",
      "          'infections and 42,000 deaths ( [8] , March 31st). COVID-19 has a '\n",
      "          'basic reproductive number, R 0 , currently estimated in the region '\n",
      "          'of 2.5 -3 [5] . Social distance and general quarantine measures can '\n",
      "          'reduce R 0 temporarily, but not permanently. For R 0 = 3, left '\n",
      "          'unchecked COVID-19 can be expected to infect more than 90% of our '\n",
      "          'community, with 30% of the population infected at the epidemic '\n",
      "          'peak. Even with significant quarantine measures in place the '\n",
      "          'population will not reach \"herd immunity\" to this virus until 2/3 '\n",
      "          'of the population has gained resistance-either through vaccination, '\n",
      "          'or infection and subsequent recovery.'},\n",
      " {'cite_spans': [{...}, {...}, {...}, {...}],\n",
      "  'ref_spans': [{...}],\n",
      "  'section': 'Introduction',\n",
      "  'text': 'In order to place these numbers in a concrete context, a recent '\n",
      "          'survey in New Zealand indicated that the country had a total of 520 '\n",
      "          \"ventilator machines [7] . Given the country's demographics (see \"\n",
      "          'table 1) , and current estimates of 1 Table 1 : Here we provide '\n",
      "          'demographic data for New Zealand [2] , along with risk of ICU '\n",
      "          'admission per infection for each age group [1] . Finally we give '\n",
      "          'the expected number of ICU admissions, assuming 2/3 rd s of each '\n",
      "          'age category becomes infected over the course of the epidemic-the '\n",
      "          'minimum required to reach herd immunity.'}]\n"
     ]
    }
   ],
   "source": [
    "print(\"body_text content:\")\n",
    "pprint(file['body_text'][:2], depth=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the grouped section titles are for the example above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Introduction',\n",
      " 'Targeted Quarantine and Release',\n",
      " 'The Model',\n",
      " 'Results',\n",
      " '6',\n",
      " '7',\n",
      " 'Logistics',\n",
      " 'Assumptions to be Investigated',\n",
      " 'Opportunities',\n",
      " 'Conclusions',\n",
      " '9',\n",
      " '10',\n",
      " '11']\n"
     ]
    }
   ],
   "source": [
    "texts = [(di['section'], di['text']) for di in file['body_text']]\n",
    "texts_di = {di['section']: \"\" for di in file['body_text']}\n",
    "for section, text in texts:\n",
    "    texts_di[section] += text\n",
    "\n",
    "pprint(list(texts_di.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example shows what the final result looks like, after we format each section title with its content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Introduction\n",
      "\n",
      "COVID-19, initially observed/detected in Hubei province of China during December 2019, has since spread to all but a handful countries, causing (as of the time of writing) an estimated 855,000 infections and 42,000 deaths ( [8] , March 31st). COVID-19 has a basic reproductive number, R 0 , currently estimated in the region of 2.5 -3 [5] . Social distance and general quarantine measures can reduce R 0 temporarily, but not permanently. For R 0 = 3, left unchecked COVID-19 can be expected to infect more than 90% of our community, with 30% of the population infected at the epidemic peak. Even with significant quarantine measures in place the population will not reach \"herd immunity\" to this virus until 2/3 of the population has gained resistance-either through vaccination, or infection and subsequent recovery.In order to place these numbers in a concrete context, a recent survey in New Zealand indicated that the country had a total of 520 ventilator machines [7] . Given the country's demographics (see table 1) , and current estimates of 1 Table 1 : Here we provide demographic data for New Zealand [2] , along with risk of ICU admission per infection for each age group [1] . Finally we give the expected number of ICU admissions, assuming 2/3 rd s of each age category becomes infected over the course of the epidemic-the minimum required to reach herd immunity.Age This is 15 times more demand than could be accommodated in the expected 4 month span of an unmitigated epidemic. The details of this calculation may vary from country to county, but the final conclusion is ubiquitous -hospitals are not prepared for this disease. Efforts to \"flatten the curve\" will need to reduce the epidemic peak not merely by a factor of two, but instead by an order of magnitude or more. Even in the most optimistic scenarios, for the most well equipped countries, such efforts must be maintained for years on end.Societal lockdown may be effective at eradicating COVID-19 locally, but when lockdown is complete a large susceptible population will remain; if the virus is later re-introduced, as expected in our globalized and interconnected world, a new epidemic is likely to occur. While buying time allows for manufacturing of new medical equipment, and further scientific investigations, such efforts can not be maintained indefinitely. For this reason it proves necessary to discuss not just what quarantine measures are needed, but also how society might return to normal, and over what time frame this can be achieved.\n",
      "\n",
      "Targeted Quarantine and Release\n",
      "\n",
      "As has been observed in South Korea [4] , death rate is tightly correlated with age. While deaths in younger age categories are observed, a recent report from Italy [3] indicates that the vast majority of deaths occur in patients with known pathologies. It should thus be possible to predict who is most at risk with high 2 . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv \n"
     ]
    }
   ],
   "source": [
    "body = \"\"\n",
    "\n",
    "for section, text in texts_di.items():\n",
    "    body += section\n",
    "    body += \"\\n\\n\"\n",
    "    body += text\n",
    "    body += \"\\n\\n\"\n",
    "\n",
    "print(body[:3000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function below lets you display the body text in one line (unhide to see exactly the same as above):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_kg_hide-output": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Introduction\n",
      "\n",
      "COVID-19, initially observed/detected in Hubei province of China during December 2019, has since spread to all but a handful countries, causing (as of the time of writing) an estimated 855,000 infections and 42,000 deaths ( [8] , March 31st). COVID-19 has a basic reproductive number, R 0 , currently estimated in the region of 2.5 -3 [5] . Social distance and general quarantine measures can reduce R 0 temporarily, but not permanently. For R 0 = 3, left unchecked COVID-19 can be expected to infect more than 90% of our community, with 30% of the population infected at the epidemic peak. Even with significant quarantine measures in place the population will not reach \"herd immunity\" to this virus until 2/3 of the population has gained resistance-either through vaccination, or infection and subsequent recovery.In order to place these numbers in a concrete context, a recent survey in New Zealand indicated that the country had a total of 520 ventilator machines [7] . Given the country's demographics (see table 1) , and current estimates of 1 Table 1 : Here we provide demographic data for New Zealand [2] , along with risk of ICU admission per infection for each age group [1] . Finally we give the expected number of ICU admissions, assuming 2/3 rd s of each age category becomes infected over the course of the epidemic-the minimum required to reach herd immunity.Age This is 15 times more demand than could be accommodated in the expected 4 month span of an unmitigated epidemic. The details of this calculation may vary from country to county, but the final conclusion is ubiquitous -hospitals are not prepared for this disease. Efforts to \"flatten the curve\" will need to reduce the epidemic peak not merely by a factor of two, but instead by an order of magnitude or more. Even in the most optimistic scenarios, for the most well equipped countries, such efforts must be maintained for years on end.Societal lockdown may be effective at eradicating COVID-19 locally, but when lockdown is complete a large susceptible population will remain; if the virus is later re-introduced, as expected in our globalized and interconnected world, a new epidemic is likely to occur. While buying time allows for manufacturing of new medical equipment, and further scientific investigations, such efforts can not be maintained indefinitely. For this reason it proves necessary to discuss not just what quarantine measures are needed, but also how society might return to normal, and over what time frame this can be achieved.\n",
      "\n",
      "Targeted Quarantine and Release\n",
      "\n",
      "As has been observed in South Korea [4] , death rate is tightly correlated with age. While deaths in younger age categories are observed, a recent report from Italy [3] indicates that the vast majority of deaths occur in patients with known pathologies. It should thus be possible to predict who is most at risk with high 2 . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv \n"
     ]
    }
   ],
   "source": [
    "print(format_body(file['body_text'])[:3000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Biorxiv: Metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first see what keys are contained in the `metadata` dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['title', 'authors'])\n"
     ]
    }
   ],
   "source": [
    "print(all_files[0]['metadata'].keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at each of the correspond values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Effectiveness of Targeted Quarantine for Minimising Impact of COVID-19\n"
     ]
    }
   ],
   "source": [
    "print(all_files[0]['metadata']['title'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'affiliation': {'institution': 'Carl von Ossietzky Universität Oldenburg',\n",
      "                  'laboratory': '',\n",
      "                  'location': {'country': 'Germany'}},\n",
      "  'email': '',\n",
      "  'first': 'Alastair',\n",
      "  'last': 'Jamieson-Lane',\n",
      "  'middle': [],\n",
      "  'suffix': ''},\n",
      " {'affiliation': {'institution': 'University of British Columbia',\n",
      "                  'laboratory': '',\n",
      "                  'location': {'country': 'Canada'}},\n",
      "  'email': '',\n",
      "  'first': 'Eric',\n",
      "  'last': 'Cytrnbaum',\n",
      "  'middle': [],\n",
      "  'suffix': ''}]\n"
     ]
    }
   ],
   "source": [
    "authors = all_files[0]['metadata']['authors']\n",
    "pprint(authors[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `format_name` and `format_affiliation` functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: Alastair Jamieson-Lane\n",
      "Affiliation: Carl von Ossietzky Universität Oldenburg, Germany\n",
      "\n",
      "Name: Eric Cytrnbaum\n",
      "Affiliation: University of British Columbia, Canada\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for author in authors:\n",
    "    print(\"Name:\", format_name(author))\n",
    "    print(\"Affiliation:\", format_affiliation(author['affiliation']))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's take as an example a slightly longer list of authors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'authors': [{'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Luca',\n",
      "              'last': 'Ferretti',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Chris',\n",
      "              'last': 'Wymant',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Michelle',\n",
      "              'last': 'Kendall',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Lele',\n",
      "              'last': 'Zhao',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Anel',\n",
      "              'last': 'Nurtay',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Lucie',\n",
      "              'last': 'Abeler-Dörner',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Wellcome Centre for Ethics and '\n",
      "                                            'the Humanities and Ethox Centre',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Michael',\n",
      "              'last': 'Parker',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'David',\n",
      "              'last': 'Bonsall',\n",
      "              'middle': [],\n",
      "              'suffix': ''},\n",
      "             {'affiliation': {'institution': 'University of Oxford',\n",
      "                              'laboratory': 'Li Ka Shing Centre for Health '\n",
      "                                            'Information and Discovery',\n",
      "                              'location': {...}},\n",
      "              'email': '',\n",
      "              'first': 'Christophe',\n",
      "              'last': 'Fraser',\n",
      "              'middle': [],\n",
      "              'suffix': ''}],\n",
      " 'title': 'Quantifying SARS-CoV-2 transmission suggests epidemic control with '\n",
      "          'digital contact tracing'}\n"
     ]
    }
   ],
   "source": [
    "pprint(all_files[4]['metadata'], depth=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, I provide the function `format_authors` that let you format a list of authors to get a final string, with the optional argument of showing the affiliation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Formatting without affiliation:\n",
      "Luca Ferretti, Chris Wymant, Michelle Kendall, Lele Zhao, Anel Nurtay, Lucie Abeler-Dörner, Michael Parker, David Bonsall, Christophe Fraser\n",
      "\n",
      "Formatting with affiliation:\n",
      "Luca Ferretti (University of Oxford, Oxford, UK), Chris Wymant (University of Oxford, Oxford, UK), Michelle Kendall (University of Oxford, Oxford, UK), Lele Zhao (University of Oxford, Oxford, UK), Anel Nurtay (University of Oxford, Oxford, UK), Lucie Abeler-Dörner (University of Oxford, Oxford, UK), Michael Parker (University of Oxford, UK), David Bonsall (University of Oxford, Oxford, UK), Christophe Fraser (University of Oxford, Oxford, UK)\n"
     ]
    }
   ],
   "source": [
    "authors = all_files[4]['metadata']['authors']\n",
    "print(\"Formatting without affiliation:\")\n",
    "print(format_authors(authors, with_affiliation=False))\n",
    "print(\"\\nFormatting with affiliation:\")\n",
    "print(format_authors(authors, with_affiliation=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Biorxiv: bibliography"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the bibliography section. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'authors': [],\n",
      "  'issn': '',\n",
      "  'other_ids': {},\n",
      "  'pages': '',\n",
      "  'ref_id': 'b0',\n",
      "  'title': 'Impact of non-pharmaceutical interventions (NPIs) to reduce '\n",
      "           'COVID-19 mortality and healthcare demand',\n",
      "  'venue': '',\n",
      "  'volume': '',\n",
      "  'year': None},\n",
      " {'authors': [],\n",
      "  'issn': '',\n",
      "  'other_ids': {},\n",
      "  'pages': '',\n",
      "  'ref_id': 'b1',\n",
      "  'title': 'Place Summaries | New Zealand | Stats NZ',\n",
      "  'venue': '',\n",
      "  'volume': '',\n",
      "  'year': None}]\n"
     ]
    }
   ],
   "source": [
    "bibs = list(file['bib_entries'].values())\n",
    "pprint(bibs[:2], depth=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can reused the `format_authors` function here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "format_authors(bibs[1]['authors'], with_affiliation=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function let you format the bibliography all at once. It only extracts the title, authors, venue, year, and separate each entry of the bibliography with a `;`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand, , , None; Place Summaries | New Zealand | Stats NZ, , , None; Report sulle caratteristiche dei pazienti deceduti positivia COVID-19 inItal iaIl presente reportè basato sui dati aggiornatial 17 Marzo 2020, , , 2020; Library Catalog: www.cdc.go.kr,  Kcdc,  Kcdc, , None; The reproductive number of COVID-19 is higher compared to SARS coronavirus, Ying Liu, Albert A Gayle, Annelies Wilder-Smith, Joacim Rocklöv, Journal of Travel Medicine, 2020\n"
     ]
    }
   ],
   "source": [
    "bib_formatted = format_bib(bibs[:5])\n",
    "print(bib_formatted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Biorxiv: Generate CSV\n",
    "\n",
    "In this section, I show you how to manually generate the CSV files. As you can see, it's now super simple because of the `format_` helper functions. In the next sections, I show you have to generate them in 3 lines using the `load_files` and `generate_clean_dr` helper functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "844346def0404edfb44e2b0248539d27",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1625.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "cleaned_files = []\n",
    "\n",
    "for file in tqdm(all_files):\n",
    "    features = [\n",
    "        file['paper_id'],\n",
    "        file['metadata']['title'],\n",
    "        format_authors(file['metadata']['authors']),\n",
    "        format_authors(file['metadata']['authors'], \n",
    "                       with_affiliation=True),\n",
    "        format_body(file['abstract']),\n",
    "        format_body(file['body_text']),\n",
    "        format_bib(file['bib_entries']),\n",
    "        file['metadata']['authors'],\n",
    "        file['bib_entries']\n",
    "    ]\n",
    "    \n",
    "    cleaned_files.append(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>paper_id</th>\n",
       "      <th>title</th>\n",
       "      <th>authors</th>\n",
       "      <th>affiliations</th>\n",
       "      <th>abstract</th>\n",
       "      <th>text</th>\n",
       "      <th>bibliography</th>\n",
       "      <th>raw_authors</th>\n",
       "      <th>raw_bibliography</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bbf09194127619f57b3ddf5daf684593a5831367</td>\n",
       "      <td>The Effectiveness of Targeted Quarantine for M...</td>\n",
       "      <td>Alastair Jamieson-Lane, Eric Cytrnbaum</td>\n",
       "      <td>Alastair Jamieson-Lane (Carl von Ossietzky Uni...</td>\n",
       "      <td>Abstract\\n\\nWe model the extent to which age t...</td>\n",
       "      <td>Introduction\\n\\nCOVID-19, initially observed/d...</td>\n",
       "      <td>Impact of non-pharmaceutical interventions (NP...</td>\n",
       "      <td>[{'first': 'Alastair', 'middle': [], 'last': '...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Impact ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2a21fdd15e07c89c88e8c2f6c6ab5692568876ec</td>\n",
       "      <td>Evaluation of Group Testing for SARS-CoV-2 RNA</td>\n",
       "      <td>Nasa Sinnott-Armstrong, Daniel L Klein, Brenda...</td>\n",
       "      <td>Nasa Sinnott-Armstrong, Daniel L Klein, Brenda...</td>\n",
       "      <td>Abstract\\n\\nDuring the current COVID-19 pandem...</td>\n",
       "      <td>Introduction\\n\\nGroup testing was first descri...</td>\n",
       "      <td>In one Italian town, we showed mass testing co...</td>\n",
       "      <td>[{'first': 'Nasa', 'middle': [], 'last': 'Sinn...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'In one ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>e686d1ce1540026ecb100c09f99ed091c139b92c</td>\n",
       "      <td>Why estimating population-based case fatality ...</td>\n",
       "      <td>Lucas Böttcher, Mingtao Xia, Tom Chou</td>\n",
       "      <td>Lucas Böttcher, Mingtao Xia (UCLA, 90095-1555,...</td>\n",
       "      <td>Abstract\\n\\nDifferent ways of calculating mort...</td>\n",
       "      <td>\\n\\nDifferent ways of calculating mortality ra...</td>\n",
       "      <td>COVID-19 statistics, , , None; The Lancet, Z X...</td>\n",
       "      <td>[{'first': 'Lucas', 'middle': [], 'last': 'Böt...</td>\n",
       "      <td>{'BIBREF2': {'ref_id': 'b2', 'title': 'COVID-1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c6039f8933305c9f44a44c81a15b321b6c2848dc</td>\n",
       "      <td>Far-UVC light: A new tool to control the sprea...</td>\n",
       "      <td>David Welch, Manuela Buonanno, Veljko Grilj, I...</td>\n",
       "      <td>David Welch (Columbia University Medical Cente...</td>\n",
       "      <td>Abstract\\n\\nAirborne-mediated microbial diseas...</td>\n",
       "      <td>3\\n\\nAirborne-mediated microbial diseases repr...</td>\n",
       "      <td>Global, regional, and national life expectancy...</td>\n",
       "      <td>[{'first': 'David', 'middle': [], 'last': 'Wel...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Global,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>073d74442e2655d79b0b3f764a627ec667ad422c</td>\n",
       "      <td>Quantifying SARS-CoV-2 transmission suggests e...</td>\n",
       "      <td>Luca Ferretti, Chris Wymant, Michelle Kendall,...</td>\n",
       "      <td>Luca Ferretti (University of Oxford, Oxford, U...</td>\n",
       "      <td>Abstract\\n\\nThe newly emergent human virus SAR...</td>\n",
       "      <td>IV.\\n\\nEnvironmental transmission: transmissio...</td>\n",
       "      <td>Early Transmission Dynamics in Wuhan, China, o...</td>\n",
       "      <td>[{'first': 'Luca', 'middle': [], 'last': 'Ferr...</td>\n",
       "      <td>{'BIBREF2': {'ref_id': 'b2', 'title': 'Early T...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   paper_id  \\\n",
       "0  bbf09194127619f57b3ddf5daf684593a5831367   \n",
       "1  2a21fdd15e07c89c88e8c2f6c6ab5692568876ec   \n",
       "2  e686d1ce1540026ecb100c09f99ed091c139b92c   \n",
       "3  c6039f8933305c9f44a44c81a15b321b6c2848dc   \n",
       "4  073d74442e2655d79b0b3f764a627ec667ad422c   \n",
       "\n",
       "                                               title  \\\n",
       "0  The Effectiveness of Targeted Quarantine for M...   \n",
       "1     Evaluation of Group Testing for SARS-CoV-2 RNA   \n",
       "2  Why estimating population-based case fatality ...   \n",
       "3  Far-UVC light: A new tool to control the sprea...   \n",
       "4  Quantifying SARS-CoV-2 transmission suggests e...   \n",
       "\n",
       "                                             authors  \\\n",
       "0             Alastair Jamieson-Lane, Eric Cytrnbaum   \n",
       "1  Nasa Sinnott-Armstrong, Daniel L Klein, Brenda...   \n",
       "2              Lucas Böttcher, Mingtao Xia, Tom Chou   \n",
       "3  David Welch, Manuela Buonanno, Veljko Grilj, I...   \n",
       "4  Luca Ferretti, Chris Wymant, Michelle Kendall,...   \n",
       "\n",
       "                                        affiliations  \\\n",
       "0  Alastair Jamieson-Lane (Carl von Ossietzky Uni...   \n",
       "1  Nasa Sinnott-Armstrong, Daniel L Klein, Brenda...   \n",
       "2  Lucas Böttcher, Mingtao Xia (UCLA, 90095-1555,...   \n",
       "3  David Welch (Columbia University Medical Cente...   \n",
       "4  Luca Ferretti (University of Oxford, Oxford, U...   \n",
       "\n",
       "                                            abstract  \\\n",
       "0  Abstract\\n\\nWe model the extent to which age t...   \n",
       "1  Abstract\\n\\nDuring the current COVID-19 pandem...   \n",
       "2  Abstract\\n\\nDifferent ways of calculating mort...   \n",
       "3  Abstract\\n\\nAirborne-mediated microbial diseas...   \n",
       "4  Abstract\\n\\nThe newly emergent human virus SAR...   \n",
       "\n",
       "                                                text  \\\n",
       "0  Introduction\\n\\nCOVID-19, initially observed/d...   \n",
       "1  Introduction\\n\\nGroup testing was first descri...   \n",
       "2  \\n\\nDifferent ways of calculating mortality ra...   \n",
       "3  3\\n\\nAirborne-mediated microbial diseases repr...   \n",
       "4  IV.\\n\\nEnvironmental transmission: transmissio...   \n",
       "\n",
       "                                        bibliography  \\\n",
       "0  Impact of non-pharmaceutical interventions (NP...   \n",
       "1  In one Italian town, we showed mass testing co...   \n",
       "2  COVID-19 statistics, , , None; The Lancet, Z X...   \n",
       "3  Global, regional, and national life expectancy...   \n",
       "4  Early Transmission Dynamics in Wuhan, China, o...   \n",
       "\n",
       "                                         raw_authors  \\\n",
       "0  [{'first': 'Alastair', 'middle': [], 'last': '...   \n",
       "1  [{'first': 'Nasa', 'middle': [], 'last': 'Sinn...   \n",
       "2  [{'first': 'Lucas', 'middle': [], 'last': 'Böt...   \n",
       "3  [{'first': 'David', 'middle': [], 'last': 'Wel...   \n",
       "4  [{'first': 'Luca', 'middle': [], 'last': 'Ferr...   \n",
       "\n",
       "                                    raw_bibliography  \n",
       "0  {'BIBREF0': {'ref_id': 'b0', 'title': 'Impact ...  \n",
       "1  {'BIBREF0': {'ref_id': 'b0', 'title': 'In one ...  \n",
       "2  {'BIBREF2': {'ref_id': 'b2', 'title': 'COVID-1...  \n",
       "3  {'BIBREF0': {'ref_id': 'b0', 'title': 'Global,...  \n",
       "4  {'BIBREF2': {'ref_id': 'b2', 'title': 'Early T...  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = [\n",
    "    'paper_id', \n",
    "    'title', \n",
    "    'authors',\n",
    "    'affiliations', \n",
    "    'abstract', \n",
    "    'text', \n",
    "    'bibliography',\n",
    "    'raw_authors',\n",
    "    'raw_bibliography'\n",
    "]\n",
    "\n",
    "clean_df = pd.DataFrame(cleaned_files, columns=col_names)\n",
    "clean_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df.to_csv('biorxiv_clean.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate CSV: Custom (PMC), Commercial, Non-commercial licenses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a9eb0c2d828f4dc1afc802fb6e5ba368",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=26505.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "faad4990748d43ed833fb1e2d85e5fe3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=26505.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>paper_id</th>\n",
       "      <th>title</th>\n",
       "      <th>authors</th>\n",
       "      <th>affiliations</th>\n",
       "      <th>abstract</th>\n",
       "      <th>text</th>\n",
       "      <th>bibliography</th>\n",
       "      <th>raw_authors</th>\n",
       "      <th>raw_bibliography</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14572a7a9b3e92b960d92d9755979eb94c448bb5</td>\n",
       "      <td>Immune Parameters of Dry Cows Fed Mannan Oligo...</td>\n",
       "      <td>S T Franklin, M C Newman, K E Newman, K I Meek</td>\n",
       "      <td>S T Franklin (University of Kentucky, 40546-02...</td>\n",
       "      <td>Abstract\\n\\nThe objective of this study was to...</td>\n",
       "      <td>INTRODUCTION\\n\\nThe periparturient period is a...</td>\n",
       "      <td>Immune response of pregnant heifers and cows t...</td>\n",
       "      <td>[{'first': 'S', 'middle': ['T'], 'last': 'Fran...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Immune ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bb790e8366da63c4f5e2d64fa7bbd5673b93063c</td>\n",
       "      <td>Discontinuous Transcription or RNA Processing ...</td>\n",
       "      <td>Beate Schwer, Paolo Vista, Jan C Vos, Hendrik ...</td>\n",
       "      <td>Beate Schwer, Paolo Vista, Jan C Vos, Hendrik ...</td>\n",
       "      <td></td>\n",
       "      <td>Discontinuous\\n\\nTranscription or RNA Processi...</td>\n",
       "      <td>Poly (riboadenylic acid) preferentially inhibi...</td>\n",
       "      <td>[{'first': 'Beate', 'middle': [], 'last': 'Sch...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Poly (r...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>24f204ce5a1a4d752dc9ea7525082d225caed8b3</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Letter to the Editor\\n\\nThe non-contact handhe...</td>\n",
       "      <td>Novel coronavirus is putting the whole world o...</td>\n",
       "      <td>[]</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Novel c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f5bc62a289ef384131f592ec3a8852545304513a</td>\n",
       "      <td>Pediatric Natural Deaths 30</td>\n",
       "      <td>Elizabeth C Burton, Nicole A Singer</td>\n",
       "      <td>Elizabeth C Burton (Johns Hopkins University S...</td>\n",
       "      <td></td>\n",
       "      <td>Introduction\\n\\nWorldwide, the leading causes ...</td>\n",
       "      <td>In athletes who experienced sudden death or in...</td>\n",
       "      <td>[{'first': 'Elizabeth', 'middle': ['C'], 'last...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'In athl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ab78a42c688ac199a2d5669e42ee4c39ff0df2b8</td>\n",
       "      <td>A real-time convective PCR machine in a capill...</td>\n",
       "      <td>Yi-Fan Hsieh, Da-Sheng Lee, Ping-Hei Chen, Sha...</td>\n",
       "      <td>Yi-Fan Hsieh (National Taiwan University, 106,...</td>\n",
       "      <td>Abstract\\n\\nThis research reports the design, ...</td>\n",
       "      <td>Introduction\\n\\nMullis et al. developed the po...</td>\n",
       "      <td>The Polymerase Chain Reaction, K B Mullis, F F...</td>\n",
       "      <td>[{'first': 'Yi-Fan', 'middle': [], 'last': 'Hs...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'The Pol...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   paper_id  \\\n",
       "0  14572a7a9b3e92b960d92d9755979eb94c448bb5   \n",
       "1  bb790e8366da63c4f5e2d64fa7bbd5673b93063c   \n",
       "2  24f204ce5a1a4d752dc9ea7525082d225caed8b3   \n",
       "3  f5bc62a289ef384131f592ec3a8852545304513a   \n",
       "4  ab78a42c688ac199a2d5669e42ee4c39ff0df2b8   \n",
       "\n",
       "                                               title  \\\n",
       "0  Immune Parameters of Dry Cows Fed Mannan Oligo...   \n",
       "1  Discontinuous Transcription or RNA Processing ...   \n",
       "2                                                      \n",
       "3                        Pediatric Natural Deaths 30   \n",
       "4  A real-time convective PCR machine in a capill...   \n",
       "\n",
       "                                             authors  \\\n",
       "0     S T Franklin, M C Newman, K E Newman, K I Meek   \n",
       "1  Beate Schwer, Paolo Vista, Jan C Vos, Hendrik ...   \n",
       "2                                                      \n",
       "3                Elizabeth C Burton, Nicole A Singer   \n",
       "4  Yi-Fan Hsieh, Da-Sheng Lee, Ping-Hei Chen, Sha...   \n",
       "\n",
       "                                        affiliations  \\\n",
       "0  S T Franklin (University of Kentucky, 40546-02...   \n",
       "1  Beate Schwer, Paolo Vista, Jan C Vos, Hendrik ...   \n",
       "2                                                      \n",
       "3  Elizabeth C Burton (Johns Hopkins University S...   \n",
       "4  Yi-Fan Hsieh (National Taiwan University, 106,...   \n",
       "\n",
       "                                            abstract  \\\n",
       "0  Abstract\\n\\nThe objective of this study was to...   \n",
       "1                                                      \n",
       "2                                                      \n",
       "3                                                      \n",
       "4  Abstract\\n\\nThis research reports the design, ...   \n",
       "\n",
       "                                                text  \\\n",
       "0  INTRODUCTION\\n\\nThe periparturient period is a...   \n",
       "1  Discontinuous\\n\\nTranscription or RNA Processi...   \n",
       "2  Letter to the Editor\\n\\nThe non-contact handhe...   \n",
       "3  Introduction\\n\\nWorldwide, the leading causes ...   \n",
       "4  Introduction\\n\\nMullis et al. developed the po...   \n",
       "\n",
       "                                        bibliography  \\\n",
       "0  Immune response of pregnant heifers and cows t...   \n",
       "1  Poly (riboadenylic acid) preferentially inhibi...   \n",
       "2  Novel coronavirus is putting the whole world o...   \n",
       "3  In athletes who experienced sudden death or in...   \n",
       "4  The Polymerase Chain Reaction, K B Mullis, F F...   \n",
       "\n",
       "                                         raw_authors  \\\n",
       "0  [{'first': 'S', 'middle': ['T'], 'last': 'Fran...   \n",
       "1  [{'first': 'Beate', 'middle': [], 'last': 'Sch...   \n",
       "2                                                 []   \n",
       "3  [{'first': 'Elizabeth', 'middle': ['C'], 'last...   \n",
       "4  [{'first': 'Yi-Fan', 'middle': [], 'last': 'Hs...   \n",
       "\n",
       "                                    raw_bibliography  \n",
       "0  {'BIBREF0': {'ref_id': 'b0', 'title': 'Immune ...  \n",
       "1  {'BIBREF0': {'ref_id': 'b0', 'title': 'Poly (r...  \n",
       "2  {'BIBREF0': {'ref_id': 'b0', 'title': 'Novel c...  \n",
       "3  {'BIBREF0': {'ref_id': 'b0', 'title': 'In athl...  \n",
       "4  {'BIBREF0': {'ref_id': 'b0', 'title': 'The Pol...  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmc_dir = '/kaggle/input/CORD-19-research-challenge/custom_license/custom_license/pdf_json/'\n",
    "pmc_files = load_files(pmc_dir)\n",
    "pmc_df = generate_clean_df(pmc_files)\n",
    "pmc_df.to_csv('clean_pmc.csv', index=False)\n",
    "pmc_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "936d2fb5255349c3ae772b8a9cff564d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=9524.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1c7e27ae6527451c96547fa51cf88028",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=9524.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>paper_id</th>\n",
       "      <th>title</th>\n",
       "      <th>authors</th>\n",
       "      <th>affiliations</th>\n",
       "      <th>abstract</th>\n",
       "      <th>text</th>\n",
       "      <th>bibliography</th>\n",
       "      <th>raw_authors</th>\n",
       "      <th>raw_bibliography</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>37cf1eb9be84f2b178b4d60de7ba9520deccf453</td>\n",
       "      <td>Processing of the SARS-CoV pp1a/ab nsp7-10 region</td>\n",
       "      <td>Boris Krichel, Sven Falke, Rolf Hilgenfeld, La...</td>\n",
       "      <td>Boris Krichel (Leibniz Institute for Experimen...</td>\n",
       "      <td>Abstract\\n\\nSevere acute respiratory syndrome ...</td>\n",
       "      <td>Introduction\\n\\nThe discovery of severe acute ...</td>\n",
       "      <td>Identification of a novel coronavirus in patie...</td>\n",
       "      <td>[{'first': 'Boris', 'middle': [], 'last': 'Kri...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Identif...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8d095d0275e474dbb9d9b63a75591ff2c0667d73</td>\n",
       "      <td>Evidence of Recombination and Genetic Diversit...</td>\n",
       "      <td>Ting Huang, Wei Wang, Mael Bessaud, Peijun Ren...</td>\n",
       "      <td>Ting Huang (Unit of Emerging Viruses, Shanghai...</td>\n",
       "      <td>Abstract\\n\\nBackground: Human rhinoviruses (HR...</td>\n",
       "      <td>Introduction\\n\\nHuman rhinoviruses (HRVs) are ...</td>\n",
       "      <td>Rhinovirus and the lower respiratory tract, F ...</td>\n",
       "      <td>[{'first': 'Ting', 'middle': [], 'last': 'Huan...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Rhinovi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>43918c75d7b7d3f9402b1342dfea1e19eb2bd909</td>\n",
       "      <td>Antibacterial Properties of Visible-Light-Resp...</td>\n",
       "      <td>Der-Shan Sun, Jyh-Hwa Kau, Hsin-Hsien Huang, Y...</td>\n",
       "      <td>Der-Shan Sun (Tzu-Chi University, 97004, Huali...</td>\n",
       "      <td>Abstract\\n\\nThe bactericidal activity of conve...</td>\n",
       "      <td>Introduction\\n\\nAnthrax is a life-threatening ...</td>\n",
       "      <td>Anthrax in humans and animals, , Anthrax in Hu...</td>\n",
       "      <td>[{'first': 'Der-Shan', 'middle': [], 'last': '...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Anthrax...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4e65fbf4e2a747fb763730223d749a1a1288ca99</td>\n",
       "      <td>Revista da Sociedade Brasileira de Medicina Tr...</td>\n",
       "      <td>Sebastian Vernal , Sebastian Vernal, Yuri Casa...</td>\n",
       "      <td>Sebastian Vernal , Sebastian Vernal, Yuri Casa...</td>\n",
       "      <td>Abstract\\n\\nTegumentary leishmaniasis (TL) dia...</td>\n",
       "      <td>INTRODUCTION\\n\\nDiagnosing American tegumentar...</td>\n",
       "      <td>Differential diagnosis of 86 cases with initia...</td>\n",
       "      <td>[{'first': 'Sebastian', 'middle': ['Vernal'], ...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Differe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>daaa53a49dc0d23015e22a150495ac657aa993d8</td>\n",
       "      <td>Transcriptional and Translational Landscape of...</td>\n",
       "      <td>Hazel Stewart, Katherine Brown, Adam M Dinan, ...</td>\n",
       "      <td>Hazel Stewart (University of Cambridge, Cambri...</td>\n",
       "      <td>Abstract\\n\\nThe genus Torovirus (subfamily Tor...</td>\n",
       "      <td>\\n\\ncattle, goats, sheep, pigs, and horses, ca...</td>\n",
       "      <td>Mesoniviridae: a proposed new family in the or...</td>\n",
       "      <td>[{'first': 'Hazel', 'middle': [], 'last': 'Ste...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Mesoniv...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   paper_id  \\\n",
       "0  37cf1eb9be84f2b178b4d60de7ba9520deccf453   \n",
       "1  8d095d0275e474dbb9d9b63a75591ff2c0667d73   \n",
       "2  43918c75d7b7d3f9402b1342dfea1e19eb2bd909   \n",
       "3  4e65fbf4e2a747fb763730223d749a1a1288ca99   \n",
       "4  daaa53a49dc0d23015e22a150495ac657aa993d8   \n",
       "\n",
       "                                               title  \\\n",
       "0  Processing of the SARS-CoV pp1a/ab nsp7-10 region   \n",
       "1  Evidence of Recombination and Genetic Diversit...   \n",
       "2  Antibacterial Properties of Visible-Light-Resp...   \n",
       "3  Revista da Sociedade Brasileira de Medicina Tr...   \n",
       "4  Transcriptional and Translational Landscape of...   \n",
       "\n",
       "                                             authors  \\\n",
       "0  Boris Krichel, Sven Falke, Rolf Hilgenfeld, La...   \n",
       "1  Ting Huang, Wei Wang, Mael Bessaud, Peijun Ren...   \n",
       "2  Der-Shan Sun, Jyh-Hwa Kau, Hsin-Hsien Huang, Y...   \n",
       "3  Sebastian Vernal , Sebastian Vernal, Yuri Casa...   \n",
       "4  Hazel Stewart, Katherine Brown, Adam M Dinan, ...   \n",
       "\n",
       "                                        affiliations  \\\n",
       "0  Boris Krichel (Leibniz Institute for Experimen...   \n",
       "1  Ting Huang (Unit of Emerging Viruses, Shanghai...   \n",
       "2  Der-Shan Sun (Tzu-Chi University, 97004, Huali...   \n",
       "3  Sebastian Vernal , Sebastian Vernal, Yuri Casa...   \n",
       "4  Hazel Stewart (University of Cambridge, Cambri...   \n",
       "\n",
       "                                            abstract  \\\n",
       "0  Abstract\\n\\nSevere acute respiratory syndrome ...   \n",
       "1  Abstract\\n\\nBackground: Human rhinoviruses (HR...   \n",
       "2  Abstract\\n\\nThe bactericidal activity of conve...   \n",
       "3  Abstract\\n\\nTegumentary leishmaniasis (TL) dia...   \n",
       "4  Abstract\\n\\nThe genus Torovirus (subfamily Tor...   \n",
       "\n",
       "                                                text  \\\n",
       "0  Introduction\\n\\nThe discovery of severe acute ...   \n",
       "1  Introduction\\n\\nHuman rhinoviruses (HRVs) are ...   \n",
       "2  Introduction\\n\\nAnthrax is a life-threatening ...   \n",
       "3  INTRODUCTION\\n\\nDiagnosing American tegumentar...   \n",
       "4  \\n\\ncattle, goats, sheep, pigs, and horses, ca...   \n",
       "\n",
       "                                        bibliography  \\\n",
       "0  Identification of a novel coronavirus in patie...   \n",
       "1  Rhinovirus and the lower respiratory tract, F ...   \n",
       "2  Anthrax in humans and animals, , Anthrax in Hu...   \n",
       "3  Differential diagnosis of 86 cases with initia...   \n",
       "4  Mesoniviridae: a proposed new family in the or...   \n",
       "\n",
       "                                         raw_authors  \\\n",
       "0  [{'first': 'Boris', 'middle': [], 'last': 'Kri...   \n",
       "1  [{'first': 'Ting', 'middle': [], 'last': 'Huan...   \n",
       "2  [{'first': 'Der-Shan', 'middle': [], 'last': '...   \n",
       "3  [{'first': 'Sebastian', 'middle': ['Vernal'], ...   \n",
       "4  [{'first': 'Hazel', 'middle': [], 'last': 'Ste...   \n",
       "\n",
       "                                    raw_bibliography  \n",
       "0  {'BIBREF0': {'ref_id': 'b0', 'title': 'Identif...  \n",
       "1  {'BIBREF0': {'ref_id': 'b0', 'title': 'Rhinovi...  \n",
       "2  {'BIBREF0': {'ref_id': 'b0', 'title': 'Anthrax...  \n",
       "3  {'BIBREF0': {'ref_id': 'b0', 'title': 'Differe...  \n",
       "4  {'BIBREF0': {'ref_id': 'b0', 'title': 'Mesoniv...  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comm_dir = '/kaggle/input/CORD-19-research-challenge/comm_use_subset/comm_use_subset/pdf_json/'\n",
    "comm_files = load_files(comm_dir)\n",
    "comm_df = generate_clean_df(comm_files)\n",
    "comm_df.to_csv('clean_comm_use.csv', index=False)\n",
    "comm_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ffa89ab0d98404499403c0900fbefc4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=2490.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7acc5f63186d474fb55bdf0c64d2dfec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=2490.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>paper_id</th>\n",
       "      <th>title</th>\n",
       "      <th>authors</th>\n",
       "      <th>affiliations</th>\n",
       "      <th>abstract</th>\n",
       "      <th>text</th>\n",
       "      <th>bibliography</th>\n",
       "      <th>raw_authors</th>\n",
       "      <th>raw_bibliography</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cd92f91038067e7a10aa27d676ce696e1e4d67ce</td>\n",
       "      <td>EXPERIMENTAL AND THERAPEUTIC MEDICINE Dimethyl...</td>\n",
       "      <td>Zhen-Hong Zhu, Wen-Qi Song, Chang-Qing Zhang, ...</td>\n",
       "      <td>Zhen-Hong Zhu (Shanghai Jiao Tong University, ...</td>\n",
       "      <td>Abstract\\n\\nMesenchymal stem cells have been w...</td>\n",
       "      <td>Introduction\\n\\nOsteonecrosis of the femoral h...</td>\n",
       "      <td>Avascular necrosis of the femoral head: Vascul...</td>\n",
       "      <td>[{'first': 'Zhen-Hong', 'middle': [], 'last': ...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Avascul...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bab279da548d8bd363acd5033e9dc54e7dbb7107</td>\n",
       "      <td>Effects of school breaks on influenza- like il...</td>\n",
       "      <td>Yanhui Chu, Zhenyu Wu, Jiayi Ji, Jingyi Sun, X...</td>\n",
       "      <td>Yanhui Chu, Zhenyu Wu (Fudan University, Shang...</td>\n",
       "      <td></td>\n",
       "      <td>INTRODUCTION\\n\\nSchoolchildren play a major ro...</td>\n",
       "      <td>Estimating household and community transmissio...</td>\n",
       "      <td>[{'first': 'Yanhui', 'middle': [], 'last': 'Ch...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'Estimat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71edbd57cdd9af956a12054932e0cbdb87ce1fea</td>\n",
       "      <td>Social Network Characteristics and Body Mass I...</td>\n",
       "      <td>Won Joon Lee, Yoosik Youm, Yumie Rhee, Yeong-R...</td>\n",
       "      <td>Won Joon Lee (Yonsei University College of Med...</td>\n",
       "      <td>Abstract\\n\\nResearch has shown that obesity ap...</td>\n",
       "      <td>INTRODUCTION\\n\\nThe study of the effects of so...</td>\n",
       "      <td>The contribution of the social environment to ...</td>\n",
       "      <td>[{'first': 'Won', 'middle': ['Joon'], 'last': ...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'The con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2dfdbf2d6b77426866feaf93486327d372fd27c7</td>\n",
       "      <td>CLINICAL EXPERIMENTAL VACCINE RESEARCH</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>\\n\\nThere may be many reasons for the signific...</td>\n",
       "      <td>A short history of vaccination, S L Plotkin, S...</td>\n",
       "      <td>[]</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'A short...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0afa3ea846396533c7ca515968abcfea3f895082</td>\n",
       "      <td>Bone Marrow Dendritic Cells from Mice with an ...</td>\n",
       "      <td>Stacey L Burgess, Erica Buonomo, Maureen Carey...</td>\n",
       "      <td>Stacey L Burgess (Johns Hopkins Bloomberg Scho...</td>\n",
       "      <td>Abstract\\n\\nThere is an emerging paradigm that...</td>\n",
       "      <td>\\n\\nport neutrophil infiltration in inflammato...</td>\n",
       "      <td>WHO/PAHO informal consultation on intestinal p...</td>\n",
       "      <td>[{'first': 'Stacey', 'middle': ['L'], 'last': ...</td>\n",
       "      <td>{'BIBREF0': {'ref_id': 'b0', 'title': 'WHO/PAH...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   paper_id  \\\n",
       "0  cd92f91038067e7a10aa27d676ce696e1e4d67ce   \n",
       "1  bab279da548d8bd363acd5033e9dc54e7dbb7107   \n",
       "2  71edbd57cdd9af956a12054932e0cbdb87ce1fea   \n",
       "3  2dfdbf2d6b77426866feaf93486327d372fd27c7   \n",
       "4  0afa3ea846396533c7ca515968abcfea3f895082   \n",
       "\n",
       "                                               title  \\\n",
       "0  EXPERIMENTAL AND THERAPEUTIC MEDICINE Dimethyl...   \n",
       "1  Effects of school breaks on influenza- like il...   \n",
       "2  Social Network Characteristics and Body Mass I...   \n",
       "3             CLINICAL EXPERIMENTAL VACCINE RESEARCH   \n",
       "4  Bone Marrow Dendritic Cells from Mice with an ...   \n",
       "\n",
       "                                             authors  \\\n",
       "0  Zhen-Hong Zhu, Wen-Qi Song, Chang-Qing Zhang, ...   \n",
       "1  Yanhui Chu, Zhenyu Wu, Jiayi Ji, Jingyi Sun, X...   \n",
       "2  Won Joon Lee, Yoosik Youm, Yumie Rhee, Yeong-R...   \n",
       "3                                                      \n",
       "4  Stacey L Burgess, Erica Buonomo, Maureen Carey...   \n",
       "\n",
       "                                        affiliations  \\\n",
       "0  Zhen-Hong Zhu (Shanghai Jiao Tong University, ...   \n",
       "1  Yanhui Chu, Zhenyu Wu (Fudan University, Shang...   \n",
       "2  Won Joon Lee (Yonsei University College of Med...   \n",
       "3                                                      \n",
       "4  Stacey L Burgess (Johns Hopkins Bloomberg Scho...   \n",
       "\n",
       "                                            abstract  \\\n",
       "0  Abstract\\n\\nMesenchymal stem cells have been w...   \n",
       "1                                                      \n",
       "2  Abstract\\n\\nResearch has shown that obesity ap...   \n",
       "3                                                      \n",
       "4  Abstract\\n\\nThere is an emerging paradigm that...   \n",
       "\n",
       "                                                text  \\\n",
       "0  Introduction\\n\\nOsteonecrosis of the femoral h...   \n",
       "1  INTRODUCTION\\n\\nSchoolchildren play a major ro...   \n",
       "2  INTRODUCTION\\n\\nThe study of the effects of so...   \n",
       "3  \\n\\nThere may be many reasons for the signific...   \n",
       "4  \\n\\nport neutrophil infiltration in inflammato...   \n",
       "\n",
       "                                        bibliography  \\\n",
       "0  Avascular necrosis of the femoral head: Vascul...   \n",
       "1  Estimating household and community transmissio...   \n",
       "2  The contribution of the social environment to ...   \n",
       "3  A short history of vaccination, S L Plotkin, S...   \n",
       "4  WHO/PAHO informal consultation on intestinal p...   \n",
       "\n",
       "                                         raw_authors  \\\n",
       "0  [{'first': 'Zhen-Hong', 'middle': [], 'last': ...   \n",
       "1  [{'first': 'Yanhui', 'middle': [], 'last': 'Ch...   \n",
       "2  [{'first': 'Won', 'middle': ['Joon'], 'last': ...   \n",
       "3                                                 []   \n",
       "4  [{'first': 'Stacey', 'middle': ['L'], 'last': ...   \n",
       "\n",
       "                                    raw_bibliography  \n",
       "0  {'BIBREF0': {'ref_id': 'b0', 'title': 'Avascul...  \n",
       "1  {'BIBREF0': {'ref_id': 'b0', 'title': 'Estimat...  \n",
       "2  {'BIBREF0': {'ref_id': 'b0', 'title': 'The con...  \n",
       "3  {'BIBREF0': {'ref_id': 'b0', 'title': 'A short...  \n",
       "4  {'BIBREF0': {'ref_id': 'b0', 'title': 'WHO/PAH...  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noncomm_dir = '/kaggle/input/CORD-19-research-challenge/noncomm_use_subset/noncomm_use_subset/pdf_json/'\n",
    "noncomm_files = load_files(noncomm_dir)\n",
    "noncomm_df = generate_clean_df(noncomm_files)\n",
    "noncomm_df.to_csv('clean_noncomm_use.csv', index=False)\n",
    "noncomm_df.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "02cb97d7f7bf49cfb1020028ce66f5dd": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "038957ad0d4c490cac05b78fab6a77ac": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_d362ac811a634b81958ffbf96e582b32",
       "placeholder": "​",
       "style": "IPY_MODEL_6f784655aeae44078fcc517c830e9e80",
       "value": " 9524/9524 [01:07&lt;00:00, 140.66it/s]"
      }
     },
     "1a81807bf2fb4cafb81f5b78492ff836": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "1adb9f12422c4170a0da7052daa2ee8d": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "1c7e27ae6527451c96547fa51cf88028": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_5f893492d2cf4316a75b9fd33e59715a",
        "IPY_MODEL_038957ad0d4c490cac05b78fab6a77ac"
       ],
       "layout": "IPY_MODEL_83b82e7f69e34aea898b2dab56af71bb"
      }
     },
     "2c0b931c7d8d4a3891ca74fa25c9a4c5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b26086540b1341caa73b94f83b01b42d",
       "placeholder": "​",
       "style": "IPY_MODEL_be0c0bf6c10742a9a9c300ee299579cb",
       "value": " 9524/9524 [00:29&lt;00:00, 321.84it/s]"
      }
     },
     "2ffa89ab0d98404499403c0900fbefc4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_a7f60605b3394189928529d277a1e502",
        "IPY_MODEL_46b68e1a49eb4e9d803b337924cd3d4f"
       ],
       "layout": "IPY_MODEL_47521e156b864b4db93f2264b894f425"
      }
     },
     "364a5e75b08b42bc90c5372d7d985785": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_487b7d24d8504a32a6527f004ff5e879",
       "max": 9524.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_c1bc904a1cb248988b5d9fd5e0e9f3af",
       "value": 9524.0
      }
     },
     "3e42c9c650dd4f40981c2c36eb4252c8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "40395bfeedcb4bd68d9ec5f2f8fb887f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_5d5a61d425d442fa8bcbef17933e46d5",
       "placeholder": "​",
       "style": "IPY_MODEL_f3d95d56f0ba44ffaa29ad3861ee098e",
       "value": " 2490/2490 [00:13&lt;00:00, 183.57it/s]"
      }
     },
     "4046635320a74b35b274da3e62e6ca8c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_f868ebc7ea1e42c8aea58fe1ccadf54b",
       "placeholder": "​",
       "style": "IPY_MODEL_1a81807bf2fb4cafb81f5b78492ff836",
       "value": " 26505/26505 [05:04&lt;00:00, 86.96it/s]"
      }
     },
     "41987cbc46ac43df8a1643dc686b0933": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "46b68e1a49eb4e9d803b337924cd3d4f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_ac2bcb96a826429f8952a328e5f3de49",
       "placeholder": "​",
       "style": "IPY_MODEL_b9d6cfe854ea4f48a4b9da60b323da15",
       "value": " 2490/2490 [00:05&lt;00:00, 458.73it/s]"
      }
     },
     "47521e156b864b4db93f2264b894f425": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "487b7d24d8504a32a6527f004ff5e879": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5a836d1995854c81b62edcebd10b3671": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5d5a61d425d442fa8bcbef17933e46d5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5f64d1f045e241c191e9263c49c52359": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5f893492d2cf4316a75b9fd33e59715a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_e165fdbb00a345eaa5b44d20810c9ad1",
       "max": 9524.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_d858d7309fd0402c918d39b9a2ca9420",
       "value": 9524.0
      }
     },
     "6a495bc7e73047859e462cb613c8d085": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "6d3d270ebca6478ab173587b0f3893b7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "6f784655aeae44078fcc517c830e9e80": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "7059568c70844d8c8b815e19557c9fe5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "72876048fe524f58b507d241fe2b782a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "7acc5f63186d474fb55bdf0c64d2dfec": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_e56eca9747b94076ad3d8a9eb9d6d8bb",
        "IPY_MODEL_40395bfeedcb4bd68d9ec5f2f8fb887f"
       ],
       "layout": "IPY_MODEL_5f64d1f045e241c191e9263c49c52359"
      }
     },
     "83b82e7f69e34aea898b2dab56af71bb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "844346def0404edfb44e2b0248539d27": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_96adc8c1489647848400e981ac1e0889",
        "IPY_MODEL_87c5ad58ffbb46659746dea8df153574"
       ],
       "layout": "IPY_MODEL_02cb97d7f7bf49cfb1020028ce66f5dd"
      }
     },
     "87c5ad58ffbb46659746dea8df153574": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_7059568c70844d8c8b815e19557c9fe5",
       "placeholder": "​",
       "style": "IPY_MODEL_a0085ce496364c6886d3400c42ebba21",
       "value": " 1625/1625 [00:10&lt;00:00, 161.01it/s]"
      }
     },
     "89d68cb84cca4e0a8ac216c952bd86b3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "936d2fb5255349c3ae772b8a9cff564d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_364a5e75b08b42bc90c5372d7d985785",
        "IPY_MODEL_2c0b931c7d8d4a3891ca74fa25c9a4c5"
       ],
       "layout": "IPY_MODEL_5a836d1995854c81b62edcebd10b3671"
      }
     },
     "9642a77cb057495cac634ed4dddd82b4": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "96adc8c1489647848400e981ac1e0889": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_e5ad04fba4fa4d3392678c0b742a6bb0",
       "max": 1625.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_a264a4c8baf743e98072d0d4155ffdb7",
       "value": 1625.0
      }
     },
     "9ce262ceb1a54b6da19668ba1368e903": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_1adb9f12422c4170a0da7052daa2ee8d",
       "max": 26505.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_41987cbc46ac43df8a1643dc686b0933",
       "value": 26505.0
      }
     },
     "a0085ce496364c6886d3400c42ebba21": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "a264a4c8baf743e98072d0d4155ffdb7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "a60376094c77443da469268b3ea286bd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_72876048fe524f58b507d241fe2b782a",
       "placeholder": "​",
       "style": "IPY_MODEL_6a495bc7e73047859e462cb613c8d085",
       "value": " 26505/26505 [02:10&lt;00:00, 202.34it/s]"
      }
     },
     "a7f60605b3394189928529d277a1e502": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_6d3d270ebca6478ab173587b0f3893b7",
       "max": 2490.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_3e42c9c650dd4f40981c2c36eb4252c8",
       "value": 2490.0
      }
     },
     "a9eb0c2d828f4dc1afc802fb6e5ba368": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_9ce262ceb1a54b6da19668ba1368e903",
        "IPY_MODEL_a60376094c77443da469268b3ea286bd"
       ],
       "layout": "IPY_MODEL_9642a77cb057495cac634ed4dddd82b4"
      }
     },
     "ac2bcb96a826429f8952a328e5f3de49": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "aedf3aabefd548d4a6c3f30261c3539e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b26086540b1341caa73b94f83b01b42d": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b9d6cfe854ea4f48a4b9da60b323da15": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "bbd12b6f4d3d4c6d8330f5476c70aaa8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_d0a887e4c69c449aa07e5a946e71f8d2",
       "max": 26505.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_fcd9d2cae6fb43ab95d2d6fd05c53c89",
       "value": 26505.0
      }
     },
     "be0c0bf6c10742a9a9c300ee299579cb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "c1bc904a1cb248988b5d9fd5e0e9f3af": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "d0a887e4c69c449aa07e5a946e71f8d2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d362ac811a634b81958ffbf96e582b32": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d858d7309fd0402c918d39b9a2ca9420": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "e165fdbb00a345eaa5b44d20810c9ad1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e56eca9747b94076ad3d8a9eb9d6d8bb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_aedf3aabefd548d4a6c3f30261c3539e",
       "max": 2490.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_e8f925f119cc4903af601603aebccf7b",
       "value": 2490.0
      }
     },
     "e5ad04fba4fa4d3392678c0b742a6bb0": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e8f925f119cc4903af601603aebccf7b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "f3d95d56f0ba44ffaa29ad3861ee098e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "f868ebc7ea1e42c8aea58fe1ccadf54b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "faad4990748d43ed833fb1e2d85e5fe3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_bbd12b6f4d3d4c6d8330f5476c70aaa8",
        "IPY_MODEL_4046635320a74b35b274da3e62e6ca8c"
       ],
       "layout": "IPY_MODEL_89d68cb84cca4e0a8ac216c952bd86b3"
      }
     },
     "fcd9d2cae6fb43ab95d2d6fd05c53c89": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
